Towards an Extrinsic Evaluation of Referring Expressions in Situated Dialogs

In the field of referring expression generation, while in the static domain both intrinsic and extrinsic evaluations have been considered, extrinsic evaluation in the dynamic domain, such as in a situated collaborative dialog, has not been discussed in depth. In a dynamic domain, a crucial problem is that referring expressions do not make sense without an appropriate preceding dialog context. It is unrealistic for an evaluation to simply show a human evaluator the whole period from the beginning of a dialog up to the time point at which a referring expression is used. Hence, to make evaluation feasible it is indispensable to determine an appropriate shorter context. In order to investigate the context necessary to understand a referring expression in a situated collaborative dialog, we carried out an experiment with 33 evaluators and a Japanese referring expression corpus. The results contribute to finding the proper contexts for extrinsic evalution in dynamic domains.